Allocation of distributed data structures

ABSTRACT

Allocating distributed data structures and managing allocation of a symmetric heap can include defining, using a processor, the symmetric heap. The symmetric heap includes a symmetric partition for each process of a partitioned global address space (PGAS) system. Each symmetric partition of the symmetric heap begins at a same starting virtual memory address and has a same global symmetric break. One process of a plurality of processes of the PGAS system is configured as an allocator process that controls allocation of blocks of memory for each symmetric partition of the symmetric heap. Using the processor executing the allocator process, isomorphic fragmentation among the symmetric partitions of the symmetric heap is maintained.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 13/917,240,filed on Jun. 13, 2013, which is fully incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under contract numberHR0011-07-9-0002 awarded by the Defense Advanced Research ProjectsAgency (DARPA). The government has certain rights in the invention.

BACKGROUND

Some modern computer programs are capable of executing as one or moreprocesses distributed across one or more communicatively linked nodes. A“node” refers to a physical data processing system such as a computer.The nodes are communicatively linked by a network. Each process has anindependent virtual memory address space called a partition. Thecollection of all partitions for the computer program is called the“Partitioned Global Address Space” (PGAS). The computer program can bereferred to as a PGAS program. Each partition typically stores differentdata.

From time to time, one process may require information from anotherprocess. The process in need of information, called the requestingprocess, issues a read request using a network transfer applicationprogramming interface (API) to copy or read bytes from the partition ofanother process called the target process. The requested bytes are sentto the requesting process in a response message and, once received, arecopied into the partition of the requesting process. A similar series ofevents occurs to write bytes from the partition of one process to thepartition of another process.

Network read and write operations are also called get and putoperations. Network read and write operations are several orders ofmagnitude slower than locally performed read and write operations. Alocally performed read or write operation is one performed by a processusing the partition belonging to that process, e.g., using local memory.For this reason, PGAS programs with performance requirements aredesigned to minimize network operations in favor of local memoryoperations.

Often, a PGAS program distributes data structures across all partitionsso that each process is able to work on local data. When necessary, datais exchanged between processes using network READ and WRITE operations.One issue with this approach is the necessity of keeping track of thelocation of data within each partition. One solution for keeping trackof data is to maintain, for each distributed data structure, a list ofvirtual base addresses pointing to the memory block used in thatpartition. One such list is maintained for each partition. As anexample, consider a system with 128 processes per node withapproximately 2̂20 total processes. On a 64-bit machine, this would takeat least: 8×128×2̂20=2̂30 bytes or 1 Gb per node per distributed datastructure. This is both expensive in terms of storage as well as time toinitialize the list of virtual memory addresses.

One approach for reducing the amount of memory needed for tracking thelocation of data structures has been to utilize a single handle per datastructure. The handle is identical on all partitions with access to thedata structure. Within each partition, a table or shared variabledirectory (SVD), maps the handle to the allocation for that handle onthe same partition. Accordingly, the same 128 processes per node over2̂20 processes on a 64-bit can be stored with only 8×128 bytes or 1kilobyte per node per distributed data structure.

To utilize such an approach, address lookups for network read and/orwrite operations must be translated from the handle to a virtual baseaddress. The translation is performed by the target process. Performingthe translation, however, interrupts the target process from performingits assigned task(s) thereby reducing performance. In some systems,network hardware is able to accelerate read and write operations whenthe virtual memory address in the target partition is known by therequesting process. This acceleration is referred to as remote directmemory access (RDMA). In general, RDMA refers to the ability of one dataprocessing system to directly access memory of another data processingsystem without involving the operating system of either data processingsystem. One of the benefits of RDMA is that the target process is notinterrupted to perform address translation. RDMA requests, however,cannot be made in a PGAS implementation that uses SVD in the generalcase because the remote virtual memory address for the distributed datastructure in the target process being accessed is not known by therequesting process. The target process is asked to perform thetranslation, thereby rendering RDMA unavailable.

BRIEF SUMMARY

One or more embodiments disclosed within this specification relate tomanaging a symmetric heap and allocating distributed data structures.

A method includes defining, using a processor, the symmetric heap toinclude a symmetric partition for each process of a partitioned globaladdress space (PGAS) system. Each symmetric partition of the symmetricheap begins at a same starting virtual memory address and has a sameglobal symmetric break. The method further includes configuring oneprocess of a plurality of processes of the PGAS system as an allocatorprocess that controls allocation of blocks of memory for each symmetricpartition of the symmetric heap and, using the processor executing theallocator process, maintaining isomorphic fragmentation among thesymmetric partitions of the symmetric heap.

A system includes a processor programmed to initiate executableoperations. The executable operations include defining, using aprocessor, the symmetric heap to include a symmetric partition for eachprocess of a PGAS system. Each symmetric partition of the symmetric heapbegins at a same starting virtual memory address and has a same globalsymmetric break. The executable operations further include configuringone process of a plurality of processes of the PGAS system as anallocator process that controls allocation of blocks of memory for eachsymmetric partition of the symmetric heap and, using the processorexecuting the allocator process, maintaining isomorphic fragmentationamong the symmetric partitions of the symmetric heap.

A computer program product includes a computer readable storage mediumhaving program code stored thereon. The program code is executable by aprocessor to perform a method. The method includes defining, using theprocessor, the symmetric heap to include a symmetric partition for eachprocess of a PGAS system. Each symmetric partition of the symmetric heapbegins at a same starting virtual memory address and has a same globalsymmetric break. The method further includes configuring one process ofa plurality of processes of the PGAS system as an allocator process thatcontrols allocation of blocks of memory for each symmetric partition ofthe symmetric heap and, using the processor executing the allocatorprocess, maintaining isomorphic fragmentation among the symmetricpartitions of the symmetric heap.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary Partitioned GlobalAddress Space (PGAS) system.

FIG. 2 is a block diagram illustrating an example of a data processingsystem used in implementing the PGAS system of FIG. 1.

FIG. 3 is a pictorial view illustrating an example of a symmetric heapfor a PGAS system.

FIG. 4 is a diagram illustrating an exemplary implementation of acollective allocation operation for a symmetric heap.

FIG. 5 is a diagram illustrating another exemplary implementation of acollective operation for a symmetric heap.

FIG. 6 is a diagram illustrating an exemplary implementation of anindependent allocation for a symmetric heap.

FIG. 7 is a diagram illustrating another exemplary implementation of anindependent allocation for a symmetric heap.

FIG. 8 is a flow chart illustrating an exemplary method of managingdistributed data structures for a PGAS system.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer-readablemedium(s) having computer-readable program code embodied, e.g., stored,thereon.

Any combination of one or more computer-readable medium(s) may beutilized. The computer-readable medium may be a computer-readable signalmedium or a computer-readable storage medium. A computer-readablestorage medium may be, for example, but is not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer-readable storage medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk drive (HDD), a solid state drive (SSD), a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), a digital versatile disc (DVD),an optical storage device, a magnetic storage device, or any suitablecombination of the foregoing. As defined herein, the term“computer-readable storage medium” means a tangible storage medium thatcontains or stores program code for use by or in connection with aninstruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signalwith computer-readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer-readable signal medium may be any computer-readable medium thatis not a computer-readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber, cable, RF, etc., or any suitable combination ofthe foregoing. Computer program code for carrying out operations foraspects of the present invention may be written in any combination ofone or more programming languages, including an object orientedprogramming language such as Java™, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer, or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer, other programmable data processing apparatus,or other devices create means for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer, other programmabledata processing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

For purposes of simplicity and clarity of illustration, elements shownin the figures have not necessarily been drawn to scale. For example,the dimensions of some of the elements may be exaggerated relative toother elements for clarity. Further, where considered appropriate,reference numbers are repeated among the figures to indicatecorresponding, analogous, or like features.

This specification relates to implementation of a computer programconfigured to execute as one or more processes distributed across one ormore communicatively linked nodes. In one aspect, the computer programis one that uses a Partitioned Global Address Space (PGAS) and, as such,is referred to as a PGAS program. A “node” refers to a physical dataprocessing system such as a computer. The nodes are communicativelylinked by a network. Thus, each node executes the PGAS program in one ormore processes. Each process within a node includes different dataitems. In accordance with the inventive arrangements disclosed herein,the PGAS program can utilize a symmetric heap.

A symmetric heap is formed of a plurality of symmetric partitions. Eachsymmetric partition is an independent virtual memory address space. Thecollection of symmetric partitions is referred to as the PGAS. As eachprocess executes, isomorphism is enforced among the symmetricpartitions. The virtual memory address of a data structure in onesymmetric partition will be the same as the virtual memory address ofthe data structure in each other symmetric partition. By enforcingisomorphism among the symmetric partitions, the benefits of remotedirect memory access (RDMA) are more efficiently exploited for purposesof data sharing among processes since the need for address transactionsis reduced or eliminated.

FIG. 1 is a block diagram illustrating an exemplary PGAS system 100.PGAS system 100 includes a plurality of nodes. Each node is illustratedas a data processing system. As such, PGAS system 100 includes dataprocessing systems 105 and 110. PGAS system 100 can include fewer ormore data processing systems. As such, the particular number of dataprocessing systems pictured is for purposes of illustration only and isnot intended as a limitation.

Each data processing system executes one or more processes of a PGASprogram. Data processing system 105 includes or executes processes 115and 120. Data processing system 110 includes or executes processes 125and 130. Each data processing system can include fewer or moreprocesses. As such, the particular number of processes pictured withineach of data processing systems 105 and 110 is for purposes ofillustration only and is not intended as a limitation.

Each process has, or is associated with, a symmetric partition. Withineach of data processing systems 105 and 110, there is a one-to-onecorrespondence between processes and symmetric partitions. Process 115has a symmetric partition 135. Process 120 has a symmetric partition140. Process 125 has a symmetric partition 145. Process 130 has asymmetric partition 150. Symmetric partitions 135, 140, 145, and 150,taken collectively, form PGAS 155.

Each of data processing systems 105 and 110 is communicatively linkedthrough network 160. Network 160 can be implemented as, or include, anyof a variety of different networks such as a WAN, a LAN, a wirelessnetwork, a mobile network, a Virtual Private Network (VPN), theInternet, the Public Switched Telephone Network (PSTN), or the like.

As discussed, each of data processing systems 105 and 110 executes aPGAS program (not shown) as the various processes illustrated. Whileeach of symmetric partitions 135, 140, 145, and 150 is isomorphic, theactual data stored within each symmetric partition can be different.Being isomorphic, each of symmetric partitions 135, 140, 145, and 150has an identical structure. In one aspect, isomorphism means that eachof symmetric partitions 135, 140, 145, and 150 is configured to start ata same virtual memory address. In addition, isomorphism means that adata structure that exists within each of symmetric partitions 135, 140,145, and 150 will be located at the same virtual memory address in eachof symmetric partitions 135, 140, 145, and 150. For example, considerthe case in which an array of 1 million entries is spread across each ofsymmetric partitions 135, 140, 145, and 150 so that each partition has250,000 entries. The array in each of symmetric partitions 135, 140,145, and 150 will have different data items forming the 250,000 entries,but will be located at the same virtual memory address in each ofsymmetric partitions 135, 140, 145, and 150.

FIG. 2 is a block diagram illustrating an example of data processingsystem 105 of PGAS system 100 of FIG. 1. The architecture illustratedand described in FIG. 2 can be used to implement each of the variousdata processing systems of PGAS system 100 of FIG. 1.

System 105 can include at least one processor 205, e.g., a centralprocessing unit, coupled to memory elements 210 through a system bus 215or other suitable circuitry. As such, system 105 can store program codewithin memory elements 210. Processor 205 executes the program codeaccessed from memory elements 210 via system bus 215 or the othersuitable circuitry.

In one aspect, system 105 is implemented as a computer or otherprogrammable data processing apparatus that is suitable for storingand/or executing program code. It should be appreciated, however, thatsystem 105 can be implemented in the form of any system including aprocessor and memory that is capable of performing and/or initiating thefunctions and/or operations described within this specification.Further, system 105 can be implemented in any of a variety of differentform factors.

Memory elements 210 include one or more physical memory devices such as,for example, a local memory (not shown) and one or more bulk storagedevices (not shown). Local memory refers to RAM or other non-persistentmemory device(s) generally used during actual execution of the programcode. Bulk storage device(s) can be implemented as a hard disk drive(HDD), solid state drive (SSD), or other persistent data storage device.System 105 also can include one or more cache memories (not shown) thatprovide temporary storage of at least some program code in order toreduce the number of times program code must be retrieved from the bulkstorage device during execution.

Input/output (I/O) devices such as a keyboard 230, a display 235, and apointing device 240 optionally can be coupled to system 105. The I/Odevices can be coupled to system 105 either directly or throughintervening I/O controllers. One or more network adapters 245 also canbe coupled to system 105 to enable system 105 to become coupled to othersystems, computer systems, remote printers, and/or remote storagedevices through intervening private or public networks. Modems, cablemodems, wireless transceivers, and Ethernet cards are examples ofdifferent types of network adapters 245 that can be used with system105.

As pictured in FIG. 2, memory elements 210 can store PGAS program 250and a symmetric partition 255. PGAS program 250, being implemented inthe form of executable program code, is executed by system 105 and, assuch, is considered an integrated part of system 105. Symmetricpartition 255, which stores data needed for execution of PGAS program250, also is an integrated part of system 105. In this regard, PGASprogram 250, symmetric partition 255, and any parameters and/orattributes utilized by PGAS program 250 and/or symmetric partition 255are functional data structures that impart functionality when employedas part of system 105 and/or PGAS system 100. As discussed, each processof PGAS program 250 has a symmetric partition for its use as local datastorage. The representation of data processing system 105 includingsymmetric partition 255 is for purposes of illustration only.

FIG. 3 is a pictorial view 300 illustrating an example of a symmetricheap for a PGAS system as described with reference to FIG. 1. Thesymmetric heap illustrated in FIG. 3 is a distributed address space fora PGAS program. Referring to graph 300, the horizontal axis representsdifferent processes of the PGAS system. The vertical axis representsvirtual memory addresses. The virtual memory address increases in thedirection of the arrow. As noted, one-to-one correspondence existsbetween processes 1-9 pictured in FIG. 3 and symmetric partitions. Inone aspect, each of processes 1-9 is located within a same node of thePGAS system. In another aspect, one or more or each of processes 1-9 canbe located in different nodes of the PGAS system.

In this example, there are nine processes. The particular number ofprocesses used in the example of FIG. 3 is for purposes of illustrationonly and is not intended as a limitation. Fewer or more processes can beincluded depending upon the PGAS system implementation. For ease ofillustration, each of processes 1-9 can be said to have a symmetricpartition also numbered 1-9. The local symmetric partition of process 1,for example, is symmetric partition 1. Each of symmetric partitions 1-9is reserved for storing distributed data structures used by the PGASprogram. Further, each of symmetric partitions 1-9 is contiguous invirtual memory.

Modern 64-bit data processing systems have a virtual memory addressspace that is several orders of magnitude larger than the availablephysical memory. The vertical block above each of processes 1-9represents the virtual memory of the process. The shaded portion of eachblock represents the portion of virtual memory that has been allocatedand used to implement a symmetric partition for that process.

In a virtual memory system, physical memory may be mapped to any virtualmemory address not already reserved for some other purpose. Twoindependent processes on the same data processing system may havedistinct physical memory with the same virtual memory address in theirrespective virtual memory address spaces. Accordingly, most of thevirtual memory address space in a 64-bit data processing system goesunused. In view of the availability of virtual memory, a region ofvirtual memory can be reserved in each node of the PGAS system and usedto implement a symmetric partition. Thus, virtual memory addresses canbe reserved systemically across processes.

As noted, symmetric partitions 1-9 are isomorphic. As such, each ofsymmetric partitions 1-9 begins at the same virtual memory address thatis referred to as the “symmetric origin.” In one aspect, the symmetricorigin is aligned on a system memory-page boundary as a common operatingsystem requirement. Each of symmetric partitions 1-9 has memory pagesmapped into memory at increasing addresses starting with the symmetricorigin. Each of symmetric partitions 1-9 has no unmapped gaps betweenmapped pages.

The lowest unmapped address greater than the symmetric origin in each ofsymmetric partitions 1-9 is called the “partition break.” In FIG. 3, thepartition break for each of symmetric partitions 1-9 is the top of theshaded region. As such, the partition break for a given process may ormay not coincide with the global symmetric break. As illustrated in FIG.3, the virtual memory address of the global symmetric break is the same,or identical, across each of processes 1-9. While the partition breakmay not be equal to the global symmetric break for a process, nopartition break is permitted to be smaller than the global symmetricbreak. Similarly, the global symmetric break is not permitted to be lessthan the symmetric origin.

For example, in processes 1, 4, and 7, the partition break is the sameas the global symmetric break. Thus, for each of processes 1, 4, and 7,the virtual memory address of the partition break is equal to thevirtual memory address of the global symmetric break. In processes 2, 3,5, 6, 8, and 9, the symmetric break is not the same as the globalsymmetric break. Thus, for each of processes 2, 3, 5, 6, 8, and 9, thevirtual memory address of the partition break is larger than the virtualmemory address for the global symmetric break.

One reason for allowing a process to have a partition break that isdifferent than the global symmetric break is that a process may allocatememory in different granularities. For example, some nodes have afeature referred to as “huge pages” in which memory can be allocated in16 MB increments. Other nodes, however, may be able to allocate memoryin smaller 64 kB increments. A node capable of allocating memory in thesmaller 64 kB increments will have a partition break that remains equalto the global symmetric break. The nodes that allocate memory in largerincrements may have a partition break that exceeds the global symmetricbreak from time to time.

When additional virtual memory is required that is not available withinthe symmetric partitions, each of the symmetric partitions is grown byincreasing the global symmetric break. In one aspect, one process issuesa grow request using a broadcast operation to each other process therebyobtaining more virtual memory. In another aspect, each processcollectively issues a grow request in unison thereby obtaining morevirtual memory. In cases where the partition break of the node is largerthan the global symmetric break by an amount at least as large as theamount by which the symmetric partition must be grown, such processes donot grow the symmetric partition.

For example, consider the case of process 6. Process 6 allocates memoryin larger increments. When processes 1, 4, and 7 are required toincrease the global symmetric break, the partition break of each ofprocesses 1, 4, and 7 also is increased as part of the growingoperation. Process 6, however, may not be required to grow symmetricpartition 6 since the partition break of symmetric partition 6 likely iseither equal to or still exceeds the increased virtual memory address towhich the global symmetric break is raised for each symmetric partition.

Consider another example, in which the symmetric origin is at virtualmemory address 0x4000 0000 0000. The global symmetric break is currentlyat virtual memory address 0x4000 1000 0000. For ease of illustration,this example uses only three symmetric partitions. Symmetric partition 1and symmetric partition 2 each has a partition break at virtual memoryaddress 0x4000 1000 000, which matches the global symmetric break.Unlike symmetric partitions 1 and 2, symmetric partition 3 has apartition break at virtual memory address 0x4000 2000 0000.

During operation, the global symmetric break is raised to virtual memoryaddress 0x4000 1800 0000. As such, a grow request is issued to eachprocess. Process 1 maps memory into the virtual memory address range0x4000 1000 0000 to 0x4000 1800 0000. Process 1 increases its partitionbreak to virtual memory address 0x4000 1800 0000. Process 2 maps memoryinto the virtual memory address range 0x4000 1000 0000 to 0x4000 20000000. Process 2 increases its partition break to virtual memory address0x4000 2000 0000. Process 3, however, already satisfies the globalsymmetric break and does nothing except for updating its stored globalsymmetric break.

In accordance with another aspect, allocation of virtual memory is adistinct process or operation than mapping memory. Referring again toFIG. 3, one process is selected as the authoritative process thatcontrols allocation. For purposes of illustration, process 1 is theauthoritative process that controls allocation. Process 1 also can bereferred to as the “allocator process.” As such, process 1 is the onlyone of processes 1-9 that maintains a record or accounting ofinformation within its symmetric partition, i.e., partition 1. Eachother one of processes 2-9 maintains an isomorphically fragmentedsymmetric partition. A span of bytes that is unused in one symmetricpartition is unused in each other symmetric partition.

The allocator process can be configured to perform various allocationfunctions or operations. For example, the allocator process can beconfigured to implement any of a variety of different, well-knownallocator algorithms and/or techniques. In one particular example, theallocator process is configured to implement the techniques described in“TLSF: a New Dynamic Memory Allocator for Real-Time Systems,” M. Masmanoet al., Real-Time Systems, ECRTS 2004, Proceedings of the 16^(th)Euromicro (2004). It should be appreciated that the aforementionedreference is but one example and that any of a variety of allocationtechniques can be used. As such, the embodiments disclosed within thisspecification are not intended to be limited by the example provided.

In controlling allocation, the allocation process observes severalrules. For example, the symmetric partition of the allocating process isfragmented in one or more contiguous blocks of memory. No two blocks ofmemory overlap. The collection of all blocks of memory covers the entireaddress space of the symmetric partition. As such, each block of memoryis either free or used. Further, each block of memory in the symmetricpartition of the allocator process exists isomorphically in each othersymmetric partition. Thus, a block of memory that is marked as used inone symmetric partition is used in each other symmetric partition.Similarly, a block of memory that is marked as free in one symmetricpartition is free in each other symmetric partition. In one aspect, thecollection of all unused blocks of memory is called, or maintained, in a“freelist” as part of the allocator process. When discussing allocationof virtual memory and managing blocks of memory for a symmetricpartition, it is understood that a block of memory refers to a block ofvirtual memory.

For purposes of illustration, however, the following is a listing offunctions and/or operations that are performed by the allocator processto manage and/or control blocks of memory.

Find-free-block: An operation that finds a block of memory in thefreelist that is at least as large as a specified or requested size fora block of memory. No block of memory is returned if the freelistincludes no block of memory that satisfies the request.

Split-block: An operation that splits an N-byte block of memory (where Nis an integer value) into two adjacent blocks of memory covering thesame address space. One block of memory is sized to accommodate arequested size. The other block of memory from the split-block operationis made up from the residue bytes.

Merge-blocks: An operation that merges two adjacent blocks of memoryinto a single larger block of memory.

Remove-block: An operation that removes a block of memory from thefreelist and marks the removed block of memory as being used.

Return-block: An operation that marks a used block of memory as unusedand inserts the now unused block of memory into the freelist.

Create-block: An operation that marks a block of memory immediatelysucceeding or following the symmetric partition as a free block ofmemory and adds the free block of memory to the freelist. This operationis performed after expanding the symmetric heap with a grow operation.

FIG. 4 is a diagram illustrating an exemplary implementation of acollective allocation operation for a symmetric heap. A “collectiveallocation” refers to an allocation process that is executedconcurrently by all processes. Each process has a pointer to the newlycreated data structure upon completion of the collective allocationoperation. For purposes of illustration, process 1 is the allocatorprocess and the PGAS system includes a total of three processes.

For example, referring to FIG. 4, each of processes 1-3 needs access toa distributed data structure with 10 kB per symmetric partition. Eachprocess begins the allocation request concurrently for an unambiguouscount of bytes per partition. Early processes wait at a “barrier” untilall processes have reached the barrier. A “barrier” is a commonprimitive used in distributed programming. A barrier is a point in thedistributed program where each process waits upon entering until allprocesses have entered the process. The barrier provides a guaranteethat no process executes commands after the barrier until every processhas executed all commands before the barrier. For example, any writes byany process before the barrier are completed before any reads by anyprocess after the barrier.

Process 1 obtains a free block of memory. For example, process 1 canimplement the “find-free block” operation that is at least as large asthe size requested, which was 10 kB. In this example, process 1successfully finds a 32 kB block of memory within symmetric partition 1(the symmetric partition of process 1). Responsive to finding a block ofmemory that is free and of sufficient size, process 1 implements the“remove block” operation to mark the block as used and remove the blockfrom the freelist. Since the size of the block of memory exceeds therequested size, process 1 can implement the “split-block” operationresulting in a 10 kB block of memory and a 22 kB block of memory. Theresidue block of memory, i.e., the 22 kB block of memory, from thesplit-block operation is returned to the freelist.

Responsive to obtaining the block of memory, process 1 broadcasts thevirtual memory address of the block of memory. Process 1 broadcasts thevirtual memory address of the 10 kB block of memory to each otherprocess, e.g., process 2 and process 3. Accordingly, each of processes1-3 now knows the location of the distributed data structure and mayfreely access the data structure. “Accessing” a data structure or memorymeans performing a read operation, a write operation, or both. Eachprocess may access its local symmetric partition independently of eachother process.

Because the location of the data structure is the same within eachsymmetric partition, each process can access data from the datastructure in the symmetric partition of another, different process.Moreover, such access can occur using RDMA without interrupting theprocess from which data is to be read or written for the purpose ofaddress translation or the like.

FIG. 5 is a diagram illustrating another exemplary implementation of acollective operation for a symmetric heap. Whereas FIG. 4 illustratesthe case in which a block of memory was available, FIG. 5 illustratesthe case in which a block of memory is not available and the symmetricheap must be grown. Again, the PGAS system includes a total of threeprocesses and process 1 is the allocator process.

In the example of FIG. 5, each of processes 1-3 needs access to adistributed data structure with 64 kB per symmetric partition. Eachprocess begins the allocation request concurrently for an unambiguouscount of bytes per partition. Early processes wait at a “barrier” untilall processes have reached the barrier.

Process 1 attempts to obtain a free block. For example, process 1implements the “find-free block” operation. In general, responsive todetermining that a sufficiently large block of memory cannot be found,process 1 increases the global symmetric break and issues a growoperation on the local symmetric partition (symmetric partition 1). Theextended bytes are added to the free virtual memory of the symmetricpartition guaranteeing the existence of a block of memory that is sizedto accommodate the request.

In this example, process 1 is unable to find a free block of memorywithin symmetric partition 1 using the freelist that is at least 64 kBin size. According, process 1 implements a “grow symmetric partition”operation on its local symmetric partition, i.e., symmetric partition 1.Accordingly, process 1 increases the virtual memory address of theglobal symmetric break. More particularly, process 1 increases thevirtual memory address of the global symmetric break by the size of theblock of memory that is needed, e.g., 64 kB in this case. Process 1inserts the newly mapped 64 kB of virtual memory into the freelist usingthe “create-block” operation. Process 1 immediately removes the 64 kBblock of memory using the “remove-block” operation. The block of memoryis appropriately sized and does not need to be split.

Process 1 then broadcasts the virtual memory address of the block ofmemory that was obtained by growing symmetric partition 1. Process 1further broadcasts the updated global symmetric break to each ofprocesses 2 and 3. Responsive to receiving an updated global symmetricbreak, each of processes 2 and 3 grow their own respective, or local,symmetric partitions to the virtual memory address of the broadcastglobal symmetric break. Process 2 grows symmetric partition 2. Process 3grows symmetric partition 3.

Further, each of processes 1-3 can enter a second barrier to ensure thatmemory has been mapped on all symmetric partitions. Each of processes1-3 now knows the location of the distributed data structure and thepages are mapped in memory.

FIG. 6 is a diagram illustrating an exemplary implementation of anindependent allocation for a symmetric heap. An independent allocationis different from a collaborative allocation and, as such, is handleddifferently. An “independent allocation” refers to an allocation requestfrom a single process without collaboration of other processes. Again,for purposes of illustration, it can be assumed that the PGAS systemincludes three processes and that process 1 is the allocator process.

In the example illustrated in FIG. 6, the requesting process is not theallocator process. As shown, process 2 requests an allocation fromprocess 1. As part of controlling allocation of memory, process 1 isconfigured to serialized received independent allocation requests. Thus,to the extent that two or more independent allocation requests arereceived concurrently or simultaneously, process 1 serializes thereceived allocation requests. Two requests are never handledconcurrently. In the case that requestor is the allocator process, therequest is serialized with requests received from other processes.Process 1 handles each request for an independent allocation as outlinedbelow.

Consider the case in which process 2 needs to create a distributed datastructure with 1 MB per symmetric partition. Process 2 sends a requestto process 1. Process 1 is interrupted to handle the request fromprocess 2. Process 1 searches for a free block of memory thataccommodates the request, e.g., is at least 1 MB in size. If a block ofmemory is found, process 1 removes the block of memory from thefreelist. The block of memory is split if required.

Process 1, for example, implements a “find-free block” operation andsuccessfully finds a free block of memory having a size of 1.2 MB.Process 1 implements the “remove-block” operation and implements the“split-block” operation. Accordingly, the residue block of memory sizedat 0.2 MB is returned to the freelist. Process 1 responds to requestor,i.e., process 2, with the virtual memory address of the 1.0 MB block ofmemory. Process 1 returns to the operation or task that was beingperformed prior to being interrupted by the request from process 2.Process 2, waiting for the response from process 1, receives theresponse specifying the virtual memory address of the block of memory.Process 2 now knows the virtual memory address of the distributed datastructure and performs any operations that may be required.

For an independent allocation, one process allocates, or requestsallocation, of memory for all processes. The other processes, however,do not know the data structure is in their symmetric partition unlessthe requesting process notifies the other processes. Still, the otherprocesses cannot allocate something in the same place because that blockof memory is marked as used by the allocator process. This maintainsisomorphism regardless of whether other processes know the particulardata structure that exists in their symmetric partition.

Referring to the example in particular, the distributed data structurethat is implemented within the block of memory within the symmetricpartition for process 2 is distributed, or split, amongst all symmetricpartitions. In this example, process 1 and process 2 know where the datastructure is located. Because the allocator process, i.e., process 1,knows the fragmentation of the symmetric partitions, the isomorphismamong all partitions is maintained. The allocator process, for example,does not allow the virtual address provided to process 2 to be used byany other process for a different data structure.

FIG. 7 is a diagram illustrating another exemplary implementation of anindependent allocation for a symmetric heap. Whereas FIG. 6 illustratesthe case in which a block of memory is available, FIG. 7 illustrates thecase where a block of memory is not available and the symmetric heapmust be grown. Again, for purposes of illustration, it can be assumedthat the PGAS system includes three processes and that process 1 is theallocator process.

In the example illustrated in FIG. 7, the requesting process is not theallocator process. As shown, process 2 requests an allocation fromprocess 1. Referring to the prior example, process 2 needs to create adistributed data structure with 1 MB per symmetric partition. Process 2sends a request to process 1. Process 1 is interrupted to handle therequest from process 2. Process 1 implements the “find-free block”operation, which does not return a block of memory of at least 1 MB insize. A block of memory of at least 1 MB is not found on the freelist.

Since no block of memory is found within partition 1 that accommodatesthe request, process 1 increases the global symmetric break to guaranteesufficient free virtual memory. Process 1 increases the global symmetricbreak and issues a grow operation to the local symmetric partition,i.e., symmetric partition 1. Process 1 creates a block from the newlymapped memory. Process 1 then broadcasts the new virtual address of theglobal symmetric break to each other process. Broadcasting the globalsymmetric break is, in effect, a command from process 1 to each otherprocess to grow their local symmetric partitions. Issuing this commandmaintains the global symmetric break invariant. Process 1, in issuingthe grow command, interrupts each other process to issue grow commandsfor their own local symmetric partitions.

When process 1 receives an acknowledgement of the growth from each otherprocess, the allocation is complete and the symmetric data structure isusable. Accordingly, responsive to the acknowledgement from each otherprocess, process 1 provides a response to the requesting process, e.g.,process 2, with the virtual memory address of the block of memory.

Referring to FIGS. 4-7, when the PGAS program begins execution, theglobal symmetric break is equal to the symmetric origin. Accordingly,the allocator thread begins growing the symmetric heap through growoperations as described when further blocks of memory are required.While the allocator process is interrupted for purposes of allocatingvirtual memory, the other processes are not interrupted for network readand/or writes.

Any single process may request a block of memory initially returned by asymmetric allocation call to be freed. Either a collective allocation oran independent allocation of a block of memory can be freed by anyprocess. If the allocator process no longer needs a block of memory, theallocator process initiates an operation to free the block of memory.For example, the allocator process implements the “return-block”operation to return the block of memory to the freelist. If a freeoperation is initiated by another process, that process sends a requestto the allocator process to free the block of memory. Accordingly, theallocator process implements the “return-block” operation for theparticular block of memory specified by the request from the otherprocess. The allocator process does not attempt to shrink the symmetricheap.

FIG. 8 is a flow chart illustrating an exemplary method 800 of managingdistributed data structures for a PGAS system. Method 800 can beperformed by a PGAS system as described within this specification. Inblock 805, a processor of the PGAS system defines a symmetric heapincluding a symmetric partition for each process of the PGAS system.Each symmetric partition of the symmetric heap begins at a same startingvirtual memory address and has a same global symmetric break. In block810, the processor configures, or designates, one process of a pluralityof processes of the PGAS system as an allocator process that controlsallocation of blocks of memory for each symmetric partition of thesymmetric heap. In block 815, the processor maintains, e.g., throughexecution of the allocator process, isomorphic fragmentation among thesymmetric partitions of the symmetric heap. Isomorphism is maintained asdescribed herein using the various operations described.

The various operations described within this specification can beimplemented as runtime routines accessible to the user. The routines canbe implemented within a language runtime of a PGAS-enabled programminglanguage. In accordance with the inventive arrangements disclosedherein, by enforcing isomorphism among the symmetric partitions of thesymmetric heap, the speed of RDMA can be more efficiently exploited.Further, the benefits of reduced storage requirements of a sharedvariable directory are maintained.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “includes,”“including,” “comprises,” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment disclosed within thisspecification. Thus, appearances of the phrases “in one embodiment,” “inan embodiment,” and similar language throughout this specification may,but do not necessarily, all refer to the same embodiment.

The term “plurality,” as used herein, is defined as two or more thantwo. The term “another,” as used herein, is defined as at least a secondor more. The term “coupled,” as used herein, is defined as connected,whether directly without any intervening elements or indirectly with oneor more intervening elements, unless otherwise indicated. Two elementsalso can be coupled mechanically, electrically, or communicativelylinked through a communication channel, pathway, network, or system. Theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill also be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms, as these terms are only used to distinguishone element from another unless stated otherwise or the contextindicates otherwise.

The term “if” may be construed to mean “when” or “upon” or “in responseto determining” or “in response to detecting,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” may be construed to mean “upon determining” or“in response to determining” or “upon detecting [the stated condition orevent]” or “in response to detecting [the stated condition or event],”depending on the context.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the embodiments disclosed within this specification havebeen presented for purposes of illustration and description, but are notintended to be exhaustive or limited to the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of theembodiments of the invention. The embodiments were chosen and describedin order to best explain the principles of the invention and thepractical application, and to enable others of ordinary skill in the artto understand the inventive arrangements for various embodiments withvarious modifications as are suited to the particular use contemplated.

What is claimed is:
 1. A method, comprising: defining, using aprocessor, a symmetric heap comprising a symmetric partition for eachprocess of a partitioned global address space (PGAS) system; whereineach symmetric partition of the symmetric heap begins at a same startingvirtual memory address and has a same global symmetric break;configuring one process of a plurality of processes of the PGAS systemas an allocator process that controls allocation of blocks of memory foreach symmetric partition of the symmetric heap; and maintaining, usingthe processor executing the allocator process, isomorphic fragmentationamong the symmetric partitions of the symmetric heap.
 2. The method ofclaim 1, wherein the symmetric partition of the allocator process is anallocator partition, the method further comprising: responsive toobtaining a block of memory of a requested size from within theallocator partition, the allocator process broadcasting a virtual memoryaddress of the block of memory to each other process.
 3. The method ofclaim 1, wherein the symmetric partition of the allocator process is anallocator partition, the method further comprising: responsive todetermining that no block of memory of a requested size is availablewithin the allocator partition, the allocator process adding virtualmemory to the allocator partition by increasing the virtual memoryaddress of the global symmetric break in the allocator partition andobtaining a block of memory from added virtual memory; and the allocatorprocess broadcasting a virtual memory address of the block of memory andthe global symmetric break to each other process.
 4. The method of claim1, further comprising: responsive to receiving, within the allocatorprocess, a request for a block of memory from a requesting process for aparticular size, obtaining a block of memory of particular size withinthe allocator partition and sending a virtual memory address of theblock of memory to the requesting process.
 5. The method of claim 1,further comprising: responsive to receiving a plurality of requests forblocks of memory within the allocator process, serializing the pluralityof requests for blocks of memory.
 6. The method of claim 1, furthercomprising: receiving a request for a block of virtual memory for aparticular size from a requesting process; responsive to determiningthat no block of memory of the particular size is available within theallocator partition, the allocator process adding virtual memory to theallocator partition by increasing the virtual memory address of theglobal symmetric break in the allocator partition and obtaining a blockof memory from added virtual memory; broadcasting the global symmetricbreak to each other process; and responsive to receiving anacknowledgement from each other process that the local symmetricpartition of each other process has grown, sending the requestingprocess a virtual memory address of the obtained block of memory.
 7. Themethod of claim 1, wherein at least one of the symmetric partitions hasa partition break that is beyond the global symmetric break.